Missing outcomes imputation in bivariate meta-analysis

Rubin (1976) introduced a foundational framework distinguishing missing data mechanisms: Missing Completely At Random (MCAR), Missing At Random (MAR), Missing Not At Random (MNAR)

These mechanisms differ from missing data patterns, which refer to the observed structure of which values are present or absent.

Patterns = “What” is missing
Mechanisms = “Why” it is missing

MCAR: Missingness is unrelated to observed or unobserved data. The missing sample is a random subsample.

\[ P(M \mid Y, \phi) = P(M \mid \phi) \]

MAR: Missingness depends on observed variables.

\[ P(M \mid Y, \phi) = P(M \mid Y_{obs}, \phi) \]

MNAR: Missingness depends on unobserved variables (or both).

\[ P(M \mid Y, \phi) = P(M \mid Y_{mis}, \phi) \]

\[ P(M \mid Y, \phi) = P(M \mid Y_{obs}, Y_{mis}, \phi) \]

• Multivariate meta-analysis with sensitivity analysis

• Pre-existing work on imputation of summary-level missing data (Carpenter, Kenward, and White 2007; Fiero, Hsu, and Bell 2017; Lu 2023; Pastore et al. 2017; Viechtbauer 2022)

Replace the unobserved outcomes with many plausible draws, then meta-analyse each imputed dataset and pool the results with Rubin’s rules.

1. Assume several donor distributions

• e.g., Uniform, Normal, Multivariate Normal

2. Repeated stochastic single imputation

• In each iteration
  1. Draw missing effect estimates, SEs, and within-study cor
  2. Fit a bivariate random-effects model (e.g., with mixmeta)
  3. Store coefficients and vcov

3. Combine with Rubin’s rules
   • Pool point estimates, between- & within-replicate variance
   • Obtain bias, SE, CI coverage across replicates
4. Evaluation
   • Bias and Coverage
   • Sensitivity analysis for the estimates under different assumptions
   • Easily interpretable plots with uncertainty intervals

• 50 studies, each with 100 participants
• Two continuous outcomes:
  • CR: Clinician Rating
  • SR: Self-Report
• Predictors: age, sex, treatment assignment
• Random effects: bivariate normal with \(\rho = 0.7\)
• Estimates obtained via Seemingly Unrelated Regression (SUR) (Zellner 1962; Riley, Tierney, and Stewart 2021)

• Missingness based on effect size estimates:
  • CR: more likely missing when treatment effect is low
  • SR: more likely missing when treatment effect is high
• Conflict resolution: keep the one furthest from the mean

1. Normal with mean \(\in {−3, 0, 3}\)
2. Truncated multivariate normal (tmvn) with mean = (1, 5)
3. Uniform distribution from −100 to 100

• Single stochastic imputation (×100)
• Bivariate random-effects meta-analysis using mixmeta (Sera 2019)
• Pooled estimates and variances via Rubin’s rules

• Bias depends on missingness rule

• Imputation from uniform distributions are conservative but imprecise

• Imputation from truncate multivariate normal best performance, but relies on true model knowledge

• In real data multiple assumptions should be tested

Cuijpers et al. (2010) collected data from 48 studies that measure depression on both a clinician rating (HRSD; Hamilton (1960)) and self-report scale (BDI; Beck et al. (1961)). The meta-analysis highlights a substantial difference between the patients’ and clinicians’ evaluations of depression, in favor of the clinician rating.

• 37 studies from Appendix A (HRSD-17 and BDI outcomes, single control group, data from METAPSY database)

• 3 studies lacked post-treatment data

• For multiple comparisons per study:

  • Retained the highest mean difference
  • BDI used to break ties

I make explicit my assumptions regarding missing data, by exploring different scenarios.

library(missmeta)
library(tmvtnorm)
library(mvtnorm)

unif11 <- function(n) runif(n, min = -11, max = 11)
norm06 <- function(n) rnorm(n, mean = 0, sd = 6)
norm66 <- function(n) rnorm(n, mean = 6, sd = 6)
normn66 <- function(n) rnorm(n, mean = -6, sd = 6)

sigma <- matrix(c(6^2,
                 CorCov(6,6,0.7), 
                 CorCov(6,6,0.7), 
                 6^2), nrow = 2)

lower <- c(-100, -100)
upper <- c(100, 100)

mtv1 <- function(n) rtmvnorm(n, mean = c(-6, -6), sigma = sigma, lower = lower, upper = upper)[,1]
mtv2 <- function(n) rtmvnorm(n, mean = c(-6, -6), sigma = sigma, lower = lower, upper = upper)[,2]

Using the function genimp we generate 500 imputed values from each distribution.

imp1 <- list(unif11, norm06, norm66, normn66, mtv1)
imp2 <- list(unif11, norm06, norm66, normn66, mtv2)

out <- mapply(function(i1, i2) {
  genimp(
    df = dmnar,
    iter = 500,
    imp1 = i1,
    imp2 = i2,
    eff1 = "EstCR",
    eff2 = "EstSR",
    se1 = "SECR",
    se2 = "SESR",
    cor = "Cor.ws",
    N = "N",
    imprho = 0.6
  )
}, i1 = imp1, i2 = imp2, SIMPLIFY = FALSE)

Afterwards, we perform the chosen analyses on each imputed dataset. The user can autonomously choose the preferred meta-analysis package to conduct multivariate meta-analysis (e.g., metaSEM, mixmeta, metafor).

library(mixmeta)

outls = unlist(out, recursive = FALSE)

res = lapply(outls, function(data) {
  theta = cbind(data$EstCR, data$EstSR)
  Sigma = cbind(data$SECR^2, CorCov(data$SECR, data$SESR, data$Cor.ws), data$SESR^2)
  
  m = mixmeta(theta, S = Sigma, method = "ml")
  s = summary(m)
  ci = confint(m)
  
  results = data.frame(
  eff1 = s$coefficients[1, 1],
  eff2 = s$coefficients[2, 1],
  se1 = s$coefficients[1, 2],
  se2 = s$coefficients[2, 2],
  cov12 = s$vcov[1, 2],
  ci.lb1 = ci[1, 1], ci.ub1 = ci[1, 2],
  ci.lb2 = ci[2, 1], ci.ub2 = ci[2, 2]
  )
  
})

res <- do.call(rbind, res)

res <- split(res,  rep(1:5, each=500))

With makepooldata we prepare the dataset to make it suitable for pooling estimates and standard errors obtained from the iterations using Rubin’s rules.

pooldata <- lapply(res, function(x) {
  makepooldata(data = x, effs = "eff", ses = "se", covs = "cov")
})

methods <- c("Uniform (-11, 11)", "Normal CR = 0, SR = 0", "Normal CR = 6, SR = 6",
               "Normal CR = -6, SR = -6", "Multivariate Normal (-6, -6)")

sumres <- mapply(function(p, label) {
  sumeth_multi(Q_mat = p$Q_mat, U_list = p$U_list, method = label)
}, p = pooldata, label = methods, SIMPLIFY = FALSE)

Via a simulation study and a real dataset we have shown the use of a flexible tool for imputing missing outcome data that:

• Forces the researcher to make explicit the assumptions on the missing outcome data

• Allows for imputation and robustness check of results under different scenarios

• Can handle MNAR data

• Is easy use and with missmeta quite straightforward to implement

• Results depend on the imputation distributions and parameters

• Performance can drop if assumptions are far from reality (SR)

• Slow at the moment if one has to test many assumptions

• Increase computational efficiency

• Extend to more than two outcomes in multivariate meta-analysis

• Explore performance against existing imputation methods and sensitivity analysis techniques